Skip to content

Add CI check that the previous release works against the new schema#3506

Merged
jonathangreen merged 8 commits into
mainfrom
chore/migration-backcompat-ci
Jul 10, 2026
Merged

Add CI check that the previous release works against the new schema#3506
jonathangreen merged 8 commits into
mainfrom
chore/migration-backcompat-ci

Conversation

@jonathangreen

@jonathangreen jonathangreen commented Jun 25, 2026

Copy link
Copy Markdown
Member

Description

Adds a CI job, backwards-compatibility-test, that verifies the previous released version of the application still works against the database schema produced by the current code. This enforces our online-migration requirement: a release's database must keep working with version N-1 of the code, because new migrations run while the previous version's webservers are still serving traffic during a deploy.

The check works by:

  1. Resolving the previous release (the latest GitHub release) and its published circ-webapp image.
  2. Initializing a fresh database with the current image, which builds the new schema.
  3. Running the previous release's database tests (pytest -m db) against that schema in external-schema mode (PALACE_TEST_DATABASE_EXTERNAL_SCHEMA), so the older code exercises the new schema. If those tests fail, the migration is not backwards compatible.

New files:

  • docker/ci/test_backwards_compatibility.sh — orchestrates the check.
  • docker/ci/test_backwards_compatibility.yml — compose overlay adding a webapp-prev service that runs the previous release's image in external-schema mode.

The push job now depends on backwards-compatibility-test so images are not tagged and pushed unless the check passes.

Motivation and Context

Our migrations are run online, while the previous version of the application is still serving traffic, so a release's database must work with version N-1 of the code. Until now nothing in CI verified that invariant — a migration that broke backwards compatibility would only be caught during a deploy. This job catches such regressions before images are pushed.

How Has This Been Tested?

The check is itself a CI job and runs as part of this PR's build against the latest published release image. The script also supports a PREV_RELEASE_IMAGE override for running it locally against a chosen previous image.

Checklist

  • I have updated the documentation accordingly.
  • All new and existing tests passed.

@claude

claude Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Claude finished @jonathangreen's task in 5m 23s —— View job


Code Review

  • Read changed files (build.yml, test_backwards_compatibility.sh, test_backwards_compatibility.yml, README.md)
  • Review CI workflow gating & skip logic
  • Review script correctness & error handling
  • Review compose overlay
  • Post review

Summary

This is a well-constructed, self-contained addition that faithfully mirrors the established test_migrations.sh / test_migrations.yml pattern, and I found no blocking issues. I checked the details that would most likely hide a bug and they all hold up:

  • Step-level skip is the correct mechanism. Putting if: !startsWith(github.ref, 'refs/tags/') on the step (not the job) means backwards-compatibility-test still reports success on tag builds, so the push job's needs is satisfied and the release proceeds. A job-level if would have marked the dependency as skipped and cascaded to skip push. Good call.
  • Image tag resolution matches the publish convention. ${prev_tag#v} strips the v from a vX.Y.Z release tag to produce X.Y.Z, which is exactly what type=semver,pattern={{version}} publishes (build.yml:163), so ghcr.io/thepalaceproject/circ-webapp:X.Y.Z resolves correctly.
  • External-schema mode is exercised correctly. initialize_instance (run in webapp) builds the schema in circ, and the overlay's PALACE_TEST_DATABASE_EXTERNAL_SCHEMA=true + PALACE_TEST_DATABASE_CREATE_DATABASE=true drive the per-worker CREATE DATABASE ... TEMPLATE circ clone path in tests/fixtures/database.py. Since the webapp init container exits (--rm) before the tests run, circ has no open connections when it's used as the template.
  • Test invocation matches the existing unit-test job. uv sync --frozen --active && pytest ... is the same pattern already used at build.yml, so the Greptile note about an old uv lacking --active isn't a regression introduced here — and the previous release is recent enough that it's a non-issue in practice.
  • Error handling is thorough: set -uo pipefail with explicit || fail guards on every critical command, the trap cleanup EXIT is registered only after the early skip/validation paths (which start nothing), and the gh release view skip path emits a visible ::warning:: rather than passing silently.

The earlier review threads (checkout pinned to @v6, the missing seam-detection step) are already resolved at head — checkout is @v7 and the seam check was removed along with the description update.

No changes requested from me.
chore/migration-backcompat-ci

@greptile-apps

greptile-apps Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a backwards-compatibility-test CI job that verifies the previous released version of the application still works against the database schema produced by the current code, enforcing the online-migration requirement that N-1 code continues to function during a rolling deploy.

  • New script (test_backwards_compatibility.sh) resolves the latest GitHub release, initialises a fresh schema via the current image, then runs the previous release's pytest -m db suite in external-schema mode against that schema.
  • New compose overlay (test_backwards_compatibility.yml) adds a webapp-prev service that inherits the full test environment from webapp but uses the previous release's image and enables PALACE_TEST_DATABASE_EXTERNAL_SCHEMA.
  • Workflow wiring: the push job now depends on backwards-compatibility-test, and both the migration-test step and the new job are conditionally skipped on tag (release) builds to avoid a deadlock where the just-published tag would resolve as the "previous" release.

Confidence Score: 5/5

Safe to merge; the new CI job is well-isolated, all critical commands have explicit error handling, and the skip logic for release builds correctly prevents a deadlock without silently bypassing the gate.

The orchestration script handles every failure path explicitly (|| fail or captured exit status), the gh exit-status capture pattern is correct given the intentional absence of set -e, the trap/cleanup placement is deliberate, and the compose overlay correctly inherits the full test environment. No changes to application code or data paths.

No files require special attention.

Important Files Changed

Filename Overview
docker/ci/test_backwards_compatibility.sh New orchestration script; error handling is explicit throughout (
docker/ci/test_backwards_compatibility.yml Compose overlay correctly extends the existing webapp service, sets PREV_RELEASE_IMAGE via env var, and enables external-schema + parallel-database-creation modes consistent with how the tests are invoked.
.github/workflows/build.yml New backwards-compatibility-test job is correctly gated on the build job, skips only its test step (not the whole job) on tag pushes to avoid deadlocking release builds, and the push job's dependency list is updated accordingly.
README.md Minor documentation reordering to clarify the relationship between PALACE_TEST_DATABASE_EXTERNAL_SCHEMA and PALACE_TEST_DATABASE_CREATE_DATABASE; no functional changes.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant GH as GitHub Actions
    participant Build as build job
    participant Compat as backwards-compatibility-test job
    participant Push as push job
    participant GHRel as GitHub Releases API
    participant Docker as Docker / Compose

    GH->>Build: trigger (push/PR)
    Build-->>Compat: image digest artifact

    Compat->>GHRel: gh release view (latest tag)
    GHRel-->>Compat: prev_tag (e.g. v1.2.3)
    Note over Compat: Skip step if tag build or no prior release

    Compat->>Docker: compose up pg, os, minio, redis
    Compat->>Docker: run webapp → initialize_instance (current schema)
    Compat->>Docker: compose pull webapp-prev (prev release image)
    Compat->>Docker: "run webapp-prev → uv sync && pytest -m db (external-schema mode)"
    Docker-->>Compat: test results

    alt Tests pass
        Compat-->>Push: success → allow push
        Push->>GH: "tag & push circ-webapp, circ-scripts, circ-exec"
    else Tests fail
        Compat-->>GH: ::error:: migration not backwards compatible
        Note over Push: push job blocked
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant GH as GitHub Actions
    participant Build as build job
    participant Compat as backwards-compatibility-test job
    participant Push as push job
    participant GHRel as GitHub Releases API
    participant Docker as Docker / Compose

    GH->>Build: trigger (push/PR)
    Build-->>Compat: image digest artifact

    Compat->>GHRel: gh release view (latest tag)
    GHRel-->>Compat: prev_tag (e.g. v1.2.3)
    Note over Compat: Skip step if tag build or no prior release

    Compat->>Docker: compose up pg, os, minio, redis
    Compat->>Docker: run webapp → initialize_instance (current schema)
    Compat->>Docker: compose pull webapp-prev (prev release image)
    Compat->>Docker: "run webapp-prev → uv sync && pytest -m db (external-schema mode)"
    Docker-->>Compat: test results

    alt Tests pass
        Compat-->>Push: success → allow push
        Push->>GH: "tag & push circ-webapp, circ-scripts, circ-exec"
    else Tests fail
        Compat-->>GH: ::error:: migration not backwards compatible
        Note over Push: push job blocked
    end
Loading

Reviews (6): Last reviewed commit: "Fix README description of the backwards-..." | Re-trigger Greptile

Comment thread .github/workflows/build.yml Outdated
Comment thread docker/ci/test_backwards_compatibility.sh
@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.46%. Comparing base (897a244) to head (2970454).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3506      +/-   ##
==========================================
+ Coverage   93.45%   93.46%   +0.01%     
==========================================
  Files         511      512       +1     
  Lines       46533    46614      +81     
  Branches     6350     6352       +2     
==========================================
+ Hits        43488    43570      +82     
+ Misses       1969     1968       -1     
  Partials     1076     1076              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jonathangreen jonathangreen marked this pull request as draft June 25, 2026 16:11
jonathangreen added a commit that referenced this pull request Jun 26, 2026
…3507)

## Description

Makes four tests robust to running serially with pytest-xdist disabled
(`-n0`). They pass under the normal xdist test run but fail under `-n0`,
which is how the backwards-compatibility CI check (#3506) runs a
previous release's database suite against the current schema.

- **Three script tests** (`test_equivalents`, `test_novelist`,
`test_search`) constructed a `Script` with no `cmd_args` and called
`do_run()`, so the script's argparse fell back to `sys.argv[1:]`. Under
xdist each worker has a clean execnet `sys.argv`, but under `-n0`
`sys.argv` is the real pytest command line. That caused argparse to
error on unrecognized arguments (`equivalents`, `novelist`), and the
leaked `-m db` marker matched `RebuildSearchIndexScript`'s
`-m/--migration` flag, sending it down the migration path so
`search_reindex.delay` was never called (`search`). Each now passes an
explicit `cmd_args=[]` to express "no arguments" independent of
`sys.argv`.
- **The OIDC logout test**
(`test_oidc_logout_initiate_no_stored_id_token`) asserted on `caplog`
without setting a capture level. Under serial execution the effective
log level is left raised by an earlier test, so the warning was not
captured and `caplog.text` was empty. It now sets the level explicitly,
matching its passing sibling
`test_oidc_logout_initiate_revocation_only`.

No production code changes — scripts intentionally default `cmd_args` to
`sys.argv` for real CLI use via `Script.run()`.

## Motivation and Context

The backwards-compatibility check added in #3506 runs the previous
release's `-m db` suite under `-n0` (external-schema mode requires
serial execution — the single shared database can't be used by parallel
xdist workers). Its first real run surfaced these four latent
test-isolation bugs, which xdist had been masking. None are
schema-compatibility failures; all four fail identically against the
previous release's own schema. Fixing them here means that once a
release containing these fixes becomes the "previous release", the check
will pass against it.

## How Has This Been Tested?

Reproduced the failures on the unfixed tests by running them under `-n0`
in the docker test environment (`tox -e py312-docker -- -n0 ...`),
observing the argparse `unrecognized arguments` errors. With the fixes
applied, all four pass under `-n0` (including with `-m db` present,
which exercises the `search` flag-collision case): `60 passed, 19
deselected`.

## Checklist

- [x] I have updated the documentation accordingly.
- [x] All new and existing tests passed.
@jonathangreen

Copy link
Copy Markdown
Member Author

This is going to fail until #3507 gets into a release, so I'll leave it in draft for now.

Add a backwards-compatibility CI job that runs the previous release's database
tests (pytest -m db) against the schema produced by the current code, exercising
the online-migration requirement that a release's database must keep working
with version N-1 of the code while its webservers are still serving traffic.

- docker/ci/test_backwards_compatibility.yml: compose overlay defining a
  webapp-prev service (the previous release's image) in external-schema mode.
- docker/ci/test_backwards_compatibility.sh: resolves the previous release from
  the latest GitHub release, builds the new schema with the current image, then
  runs the previous release's db tests against it. Skips when the previous
  release predates the external-schema seam.
- build.yml: new backwards-compatibility-test job, added to the push job's needs.

This is the second of two PRs and depends on the test seam from the first. Until
a release containing the seam becomes the previous release, the check skips.
…bility job

The previous release now ships the external-schema test seam, so the
job no longer needs to detect older images that lack it and skip.
@jonathangreen jonathangreen force-pushed the chore/migration-backcompat-ci branch from 18b6dbb to 4512a47 Compare July 7, 2026 13:42
jonathangreen and others added 5 commits July 7, 2026 10:56
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Flip the previous-release test run from serial (`-n0`) to parallel (`-n auto`)
now that external-schema mode can clone the externally-built schema into a
per-worker database. The compose overlay enables database creation
(`PALACE_TEST_DATABASE_CREATE_DATABASE=true`) so each xdist worker gets its own
clone of the schema built by the current image.

This depends on the parallel-capable external-schema fixture, which ships
separately. Because the job runs the *previous release's* baked-in test code,
this change must not merge until a release containing that fixture is the
previous release -- so keep this PR in draft until then.
The backwards-compatibility check resolves the previous-release image from
the latest GitHub release. On a release build the just-published tag is the
latest release, so the script resolves to this build's own X.Y.Z image, which
the push job has not created yet — the pull fails and the release deadlocks.

Skip both gated tests on tag builds via a step-level guard so the jobs stay
green and the push job's dependency is satisfied. The tagged commit was
already exercised by these checks on its branch/main push, so no coverage is
lost.
…resolved

The previous-release lookup ran gh with 2>/dev/null and inferred "skip" from
empty output, so any failure -- a genuine absence of releases or a transient
auth/rate-limit/network error -- was treated identically and let the gate
no-op. The backwards-compatibility check would then fail open silently, and
push would proceed without it ever running.

Let gh's error flow to the job log, check its exit status explicitly, and emit
a GitHub Actions ::warning:: on any failure before skipping. We still don't
fail the job (a blip here shouldn't block releases), but the skipped gate is
now visible at the PR/checks level instead of passing quietly.
# push, so re-running here is redundant. See backwards-compatibility-test for why
# release builds must not run these gated tests.
- name: Test migrations
if: ${{ !startsWith(github.ref, 'refs/tags/') }}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skip isn't strictly necessary, but I added it for consistency with the backwards-compatibility-test skip below. Since we shouldn't need these tests for a tagged release build.

@jonathangreen jonathangreen requested a review from a team July 7, 2026 14:43
@jonathangreen jonathangreen marked this pull request as ready for review July 7, 2026 14:43
The README attributed the backwards-compatibility CI check to the
PALACE_TEST_DATABASE_CREATE_DATABASE=false / serial path, but the check runs
with database creation enabled and pytest -n auto (parallel), matching the
external-schema-with-creation scenario described just above. Move the
attribution to that scenario and leave the create-database=false / serial
option as a general alternative.

@tdilauro tdilauro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

↔️ 🚀

@jonathangreen jonathangreen merged commit 2548bb9 into main Jul 10, 2026
25 checks passed
@jonathangreen jonathangreen deleted the chore/migration-backcompat-ci branch July 10, 2026 13:52
jonathangreen added a commit that referenced this pull request Jul 10, 2026
## Description

Fixes the `backwards-compatibility-test` job, which has been failing on
`main` since #3506 merged, with:

```
no such service: redis
##[error]Could not start service containers.
```

`docker/ci/test_backwards_compatibility.sh` asked docker compose to
start a `redis` service. #3498 renamed that compose service to `valkey`.
This renames the reference.

The service is still needed, so this is a rename rather than a removal:
`initialize_instance` itself only takes a Postgres advisory lock, but
the `webapp-prev` container inherits `PALACE_TEST_REDIS_URL:
redis://valkey:6379/3` from the `webapp` service, so the container has
to be up for the previous release's test run. The `redis://` URL scheme
and the `PALACE_*REDIS*` variable names are deliberately left alone,
consistent with #3498 — they name the protocol, not the instance.

## Motivation and Context

#3506 was branched off `main` before #3498 landed, and was never
rebased. At #3506's head commit `docker-compose.yml` still defined a
`redis` service, so the script was correct on the branch and the job
passed there. The two PRs touch disjoint files, so git merged them
without a textual conflict — the conflict was semantic, and nothing
surfaced it until the job ran on `main`.

This also means the `push` job, which now depends on
`backwards-compatibility-test`, is blocked on `main` until this lands.

## How Has This Been Tested?

Reproduced the failure and confirmed the fix locally with `docker
compose ... up --dry-run`, which validates service names without
starting containers:

- with the old `redis` argument: exit 1, `no such service: redis` — the
identical message from the failing CI log.
- with the new `valkey` argument: exit 0.

Then started the service for real and confirmed it reaches `healthy` and
that `valkey-cli ping` answers `PONG`.

I did not run the script end-to-end locally, since that requires pulling
both the current and previous release images and running the full
`pytest -m db` suite. Worth flagging for review: this fix unblocks a
step that has never actually executed, because the previous release's
test suite will now run against Valkey for the first time — on #3506's
branch it only ever ran against `redis/redis-stack-server`. I expect
that to be fine (Valkey is protocol-compatible, `main`'s own db suite
already passes against `valkey:9.0` in the tox stack, and the previous
release's Redis usage is core commands only), but this job is where that
combination gets exercised first. The job runs on non-tag refs, so this
PR's own CI is the real verification.

## Checklist

- [x] I have updated the documentation accordingly.
- [x] All new and existing tests passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants